Skip to content

Add Google Earth Engine accessor for satellite imagery (#53) - #76

Open
KarnakOza wants to merge 3 commits into
fccoelho:mainfrom
KarnakOza:feature/gee-accessor
Open

Add Google Earth Engine accessor for satellite imagery (#53)#76
KarnakOza wants to merge 3 commits into
fccoelho:mainfrom
KarnakOza:feature/gee-accessor

Conversation

@KarnakOza

@KarnakOza KarnakOza commented Jul 11, 2026

Copy link
Copy Markdown

📋 Description

Adds a Google Earth Engine (GEE) accessor providing satellite-imagery-derived
indicators for epidemiological modeling, as requested in #53.

  • Implement GoogleEarthEngineAccessor with get_ndvi, get_ndwi,
    get_built_up_index, and get_built_up_change methods, covering the
    use cases in 🛰️ Add Google Earth Engine accessor (Satellite imagery) #53 (vegetation coverage/breeding sites, urban density,
    informal settlement growth)
  • Register earthengine-api as an optional extra and add the entry point
    so the accessor is discoverable via the standard source registry
  • Add tests covering initialization, list_countries, and indicator
    sign/range sanity checks against real-world reference points
  • Raise a clear ValueError when no cloud-free Landsat imagery is
    available for a given area/date range, instead of a cryptic
    band-not-found error

🎯 Type of Change

  • 🌍 New data source

🔗 Related Issues

Closes #53

🧪 Testing

  • Tests added for new functionality
  • Tests pass locally (pytest)
  • Tested on sample data

Test commands:

pytest tests/test_accessors.py -k GoogleEarthEngine -v

📊 Data Source Details

  • Name: Google Earth Engine
  • URL: https://earthengine.google.com/
  • Geographic Coverage: Global (planetary-scale satellite imagery)
  • Update Frequency: Varies by sensor (Landsat 8 ~16 days, Sentinel-2 ~5 days)
  • Authentication Required: Yes — free Google account + Earth Engine
    registration at https://signup.earthengine.google.com/, plus a linked
    GEE Cloud Project (set via EE_PROJECT env var or project= argument)

Example usage:

from epidatasets.sources.google_earth_engine import GoogleEarthEngineAccessor

gee = GoogleEarthEngineAccessor(project="your-gee-project-id")
ndvi = gee.get_ndvi(lon=-46.63, lat=-23.55, start_date='2021-03-01', end_date='2021-03-31')

📚 Documentation

  • README.md updated
  • Docstrings added/updated
  • API documentation updated (mkdocstrings page added)

✅ Checklist

Code Quality

  • Code follows the project's style guidelines
  • Self-review of code completed
  • Code is well-commented, particularly in hard-to-understand areas

Functionality

  • Code works as expected
  • Edge cases handled appropriately (no-imagery-found case)
  • Error messages are user-friendly
  • No hardcoded credentials or sensitive data

🌍 Impact Assessment

Who will benefit:

  • Researchers modeling vector-borne disease risk using vegetation/water/
    urban-density proxies
  • Public health teams needing satellite-derived environmental indicators
    without a separate GEE integration

Breaking changes:

  • None

🔄 Testing Evidence

>>> from epidatasets.sources.google_earth_engine import GoogleEarthEngineAccessor
>>> gee = GoogleEarthEngineAccessor(project="ee-karnakoza3")
>>> gee.get_ndvi(lon=-46.63, lat=-23.55, start_date='2021-03-01', end_date='2021-03-31')
0.10143420182597954
>>> gee.get_built_up_index(lon=-46.63, lat=-23.55, start_date='2021-03-01', end_date='2021-03-31')
0.0044230300576192
>>> gee.get_built_up_index(lon=-62.5, lat=-4.0, start_date='2021-01-01', end_date='2021-12-31')
-0.1926545981623303

Validated NDBI sign convention against known land cover: positive for
dense urban core (São Paulo), negative for Amazon rainforest — confirming
the built-up index behaves correctly across contrasting terrain.

Karnak Oza and others added 2 commits July 11, 2026 23:38
- Implement GoogleEarthEngineAccessor with NDVI, NDWI, NDBI, and NDBI-change methods covering the use cases in fccoelho#53 (vegetation coverage/breeding sites, urban density, informal settlement growth)
- Register earthengine-api as an optional extra and add entry point so the accessor is discoverable via the standard source registry
- Add tests covering initialization, list_countries, and indicator sign/range sanity checks against real-world reference points
- Raise a clear ValueError when no cloud-free Landsat imagery is available for a given area/date range, instead of a cryptic band-not-found error

Closes fccoelho#53
@fccoelho

Copy link
Copy Markdown
Owner

👋 PR Review Reminder

This pull request has been open for over a week without review.

Maintainers will review soon! If you need immediate feedback, feel free to tag us.

Thanks for your contribution! 🙏

@fccoelho

fccoelho commented Aug 5, 2026

Copy link
Copy Markdown
Owner

👋 PR Review Reminder

This pull request has been open for over a week without review.

Maintainers will review soon! If you need immediate feedback, feel free to tag us.

Thanks for your contribution! 🙏

@fccoelho fccoelho left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this contribution @KarnakOza — the accessor is well-structured and closely follows the conventions of copernicus_cds.py (optional-import guard, ClassVar metadata, clear ImportError/RuntimeError messages). The index math and the no-imagery ValueError are good. However, there are two blocking issues that contradict the PR's own description, plus some correctness and test-hygiene concerns.

Blocking

1. Entry point was never registered — The PR description promises "add the entry point so the accessor is discoverable via the standard source registry", but pyproject.toml only adds the optional extra. There is no line under [project.entry-points."epidatasets.sources"] (pyproject.toml:136). Without it:

  • epidatasets.get_source("google_earth_engine") raises KeyError,
  • the source won't appear in list_sources(),
  • _registry.py:24 (_load_entry_points) won't find it.

The README still says "33 registered (via plugin registry)" — and after this PR that number is still correct, which is exactly the symptom of the missing registration.

➕ Please add, e.g.:

google_earth_engine = "epidatasets.sources.google_earth_engine:GoogleEarthEngineAccessor"

2. Not added to sources/__init__.py — Every other accessor is listed in src/epidatasets/sources/__init__.py:14-47 (under TYPE_CHECKING) and in __all__. GEE is missing from both, breaking the convention every other source follows. Please add it to both sections.

Correctness

3. Landsat C2 L2 surface-reflectance scaling is not applied (google_earth_engine.py:181). SR_B4/5/6 are raw DNs; reflectance = DN * 0.0000275 − 0.2. For normalized-difference ratios the multiplicative factor cancels, but the additive offset (−0.2) does not, biasing every index. Please either apply image.multiply(0.0000275).add(-0.2) or document explicitly why the offset is ignored.

4. Sentinel-2 / MODIS are advertised but unusedCOLLECTIONS declares sentinel2_sr and modis_vi, the docstring lists them, but every method hardcodes Landsat 8. In tropical epidemiology (the stated use case), Landsat's 16-day revisit + the 20% cloud filter means the ValueError in _landsat_composite will fire frequently. Consider a sensor="landsat8"|"sentinel2" parameter, or at least drop the unused entries to avoid false advertising.

Test hygiene

5. TestGoogleEarthEngine.test_initialization will ERROR (not skip) on any machine without EE auth (test_accessors.py:893). The accessor fixture instantiates the class, which calls ee.Initialize() in __init__. Since test_initialization lacks @requires_external_api, CI without earthengine-api installed/authenticated will report an error during setup rather than a skip. Options:

  • mark test_initialization with @requires_external_api too, or
  • add pytest.importorskip("ee") in the fixture, or
  • split into a mocked unit test + a live external test (best — matches the responses-based pattern used by TestDiseaseSh).

6. test_no_imagery_raises_clear_error is flaky (test_accessors.py:937) — it assumes a 1-day Amazon window has zero <20%-cloud Landsat scenes. A single cloud-free acquisition on 2021-03-01 would flip this. Better to monkeypatch the collection size.

Minor / nits

  • 7. logging.basicConfig(level=logging.INFO) at module import (google_earth_engine.py:43) reconfigures the root logger on every import. (Same anti-pattern exists in copernicus_cds.py:52, so it's consistent — but worth fixing repo-wide eventually.)
  • 8. _reduce_mean silently returns None on a bad band name rather than raising (google_earth_engine.py:188).
  • 9. earthengine is intentionally excluded from the all extra (pyproject.toml:105) — reasonable given interactive auth, just confirming it's deliberate.
  • 10. Modern str | None hints here vs. Optional[str] in copernicus_cds.py — minor style drift; the new style is preferable.
  • 11. README's "Optional extras: 11" update is correct, but the "Data sources: 33 registered" line was not touched (already stale — the entry-point list has 34 — but this PR doesn't fix it).

Suggested changes before merge

  • Add entry point to pyproject.toml (item 1)
  • Register in sources/__init__.py (item 2)
  • Fix test_initialization so it skips cleanly without EE (item 5)
  • Apply Landsat SR scaling or document the omission (item 3)

The accessor code itself is solid; once these are addressed the PR will deliver what its description claims and keep CI green on machines without GEE credentials.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🛰️ Add Google Earth Engine accessor (Satellite imagery)

2 participants